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1  Executive  Summary 

This  document  summarizes  the  research  conducted  between  Feb.  18,  1993  and  June 
30,  1995  on  the  project  “Distributed,  Interactive  Development  and  Monitoring  of  Trans¬ 
portation  Plans  in  Dynamic  Environments”,  supported  by  the  Rome  Laboratory  (RL)  of 
the  Air  Force  Material  Command  and  the  Defense  Advanced  Research  Projects  Agency 
(Contract  F30602-93-C-0038).  The  project  was  aimed  at  adapting  relevant  techniques 
developed  in  real-time  and  distributed  operating  systems,  and  combining  them  with 
techniques  from  real-time  and  distributed  AI,  to  support  plan  generation  in  dynamic, 
multi-agent  environments.  In  support  of  this  goal,  we  built  DIPART — the  Distributed, 
Interactive  Planner’s  Assistant  for  Real-time  Transportation  planning — a  prototype  sim¬ 
ulation  system  that  includes  a  network  of  agents,  each  of  which  assists  a  human  planner, 
and  a  simulated  dynamic  environment,  which  implements  Reece  and  Tate’s  Pacifica  NEO 
scenario  [38].  The  key  accomplishments  of  this  project  fall  into  four  main  categories: 

•  The  design  and  development  of  the  DIPART  testbed  system  (see  Section  2.) 

•  Advances  in  planning  technology  (see  Section  3),  including  the  development  of 

—  the  LCFR  strategy  for  efficient  search  control  during  plan  generation; 

—  an  effective  algorithm  for  generating  plans  in  domains  in  which  actions  have 
costs  associated  with  them,  and 

—  a  new  planning  framework  based  on  the  use  of  constraint-satisfaction  process¬ 
ing  methods. 

•  Advances  in  meta-level  reasoning  to  control  the  planning  process  in  dynamic  en¬ 
vironments  (see  Section  4),  including 

—  the  identification  of  relevant  process-scheduling  algorithms  and  their  adapta¬ 
tion  to  the  control  of  planning  in  dynamic  environments,  and 

—  the  development  of  a  framework  for  reasoning  about  which  contingencies  to 
plan  for  first,  when  time  is  limited. 

•  Advances  in  coordinating  a  distributed  planning  process  in  a  multi-agent  setting 
(see  Section  5),  including  the  development  of 

—  software  support  for  flexible  communication  strategies  among  planning  agents; 

—  a  load  balancing  mechanism  for  distributing  the  planning  effort  effectively; 

—  algorithms  for  merging  individually  formed  plans,  and 

—  a  method  for  efficiently  coordinating  the  planned  actions  of  agents  with  min¬ 
imal  communication. 
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2  The  DIPART  System 

Many  current  and  potential  AI  applications  are  intended  to  operate  in  dynamic  envi¬ 
ronments,  including  those  with  multiple  agents.  An  important  example  is  crisis  action 
planning,  which  is  typically  a  distributed  process,  involving  multiple  planners  each  tasked 
with  forming  plans  to  meet  some  subset  of  the  overall  mission  objectives.  During  plan¬ 
ning,  changes  that  occur  in  the  world  can  affect  the  quality  of  the  plans  being  created. 
When  planning  and  execution  are  interleaved,  as  they  often  must  be  in  crisis  situations, 
changes  can  also  affect  the  quality  of  plans  whose  execution  has  already  begun.  To  op¬ 
erate  in  such  environments,  standard  AI  plan-generation  technology  must  be  augmented 
with  mechanisms  for  managing  changing  information,  for  focusing  attention  when  mul¬ 
tiple  events  occur,  and  for  coordinating  with  other  planning  processes.  In  the  DIPART 
project,  we  have  been  concerned  with  the  development  and  analysis  of  such  techniques. 
Many  of  the  techniques  we  have  explored  derive  from  theoretical  work  in  real-time  AI 
and  in  related  fields,  such  as  real-time  operating  systems. 

To  support  our  research  on  plan  generation  in  dynamic,  multi-agent  environments, 
we  built  DIPART — the  Distributed,  Interactive  Planners  Assistant  for  Real-time  Trans¬ 
portation  planning.  DIPART  is  a  prototype  simulation  system  that  includes  a  network 
of  agents,  each  of  which  assists  a  human  planner,  and  a  simulated  dynamic  environment, 
which  implements  the  Pacifica  NEO  scenario  [38]. 

2.1  System  Overview 

The  DIPART  system  consists  of  a  network  of  communicating  nodes  each  assisting  a 
human  planner,  plus  a  simulated  environment  [46].  The  underlying  idea  is  that  each 
planner  has  responsibility  for  forming  and  overseeing  the  execution  of  some  set  of  plans 
that  are  carried  out  in  the  (simulated)  environment.  Each  planner  may  have  only  a 
restricted  view  of  the  environment  and  of  the  activities  of  the  other  planners;  although 
cooperation  among  the  planners  may  be  desirable,  it  is  not  automatic.  Figure  1  illus¬ 
trates  the  overall  system  architecture,  highlighting  the  internal  architecture  of  a  single 
node.  Because  each  node  performs  the  role  of  an  intelligent  assistant,  we  sometimes 
refer  to  the  nodes  as  “agent  processes” . 

The  internal  architecture  of  each  DIPART  node  is  based  on  a  generic  model  of  process 
scheduling,  similar  to  those  found  in  the  literature  on  operating  systems  [45].  Incoming 
messages  are  stored  on  a  Message  Queue  (MQ),  and  indicate  events  that  may  require 
attention.  Often  the  value  of  responding  to  a  particular  message  is  time  dependent. 
Thus,  a  mechanism  is  needed  to  determine  what  processes  should  be  invoked  in  response 
to  each  message,  and  to  schedule  the  selected  processes.  In  our  model,  the  module  that 
makes  these  decisions  is  the  Locus  of  Meta-Level  Control  (LMC);  it  is  responsible  for 
invoking  various  (object-level)  processes,  which  we  call  Reasoning  Modules  (RMs).  The 
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Other  Agents 


Figure  1:  DIPART  Architecture 

RMs  include  a  resource  estimator,  which  estimates  the  amount  of  computational  and 
external  resources  required  for  a  given  task,  a  planner,  which  computes  plans  to  achieve 
specified  goals,  and  an  execution  monitor,  which  tracks  the  performance  of  plans. 

As  shown  in  the  figure,  the  LMC  performs  its  task  by  posting  entries  in  a  Schedule 
of  Computation  (SOC).  These  entries  include  information  about  which  RM  to  invoke, 
the  input  to  that  process,  the  invocation  deadline  (i.e.,  the  time  after  which  the  system 
should  no  longer  bother  to  invoke  the  process),  and,  in  some  circumstances,  the  amount 
of  time  to  allocate  to  the  process  in  question.  A  process  controller,  called  the  Reasoning 
Modules  Manager  (RMM)  reads  entries  from  the  SOC  and  then  invokes  the  appropriate 
process.  Individual  processes  may  also  generate  messages  if  follow-on  computation  is 
needed.  A  global  database  stores  information  that  can  be  used  by  both  the  LMC  and 
the  object-level  processes. 

In  addition  to  the  agent  nodes,  DIPART  includes  a  simulator  which  has  been  tailored 
to  Pacifica  NEO  scenario,  described  below.  It  runs  as  a  separate  process  in  the  overall 
DIPART  system.  It  represents  the  “actual”  state  of  the  world;  in  contrast,  the  models 
of  the  world  kept  by  individuals  agents  may  be  limited  or  may  become  out-of-date, 
as  they  are  intended  to  represent  the  views  that  the  agents  currently  have,  given  the 
information  they  have  so  far  received.  The  simulator  is  designed  to  allow  modeling  of 
resource  allocation  to  agents. 

The  DIPART  system  has  been  implemented  on  DECStation  5000  workstations,  under 
Ultrix  4.3,  using  Allegro  Common  Lisp  and  the  Garnet  interface-development  system. 
Each  of  the  agent  nodes  runs  on  its  own  processor,  as  does  the  simulated  environment. 
Within  each  node,  the  LMC  runs  in  one  thread,  and  the  RMs  in  another.  A  com¬ 
munication  package  based  on  UDP  has  also  been  implemented  to  support  inter-node 
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communication. 


2.2  The  Pacifica  Scenario 

To  ground  our  research,  we  employ  the  Pacifica  NEO  scenario,  developed  by  Reece  and, 
Tate  for  the  RL/ARPA  Planning  Initiative  as  part  of  the  PRECiS  environment  [38]. 
This  scenario  involves  the  fictional  island  nation  of  Pacifica,  on  which  a  number  of  U.S. 
citizens  are  located.  The  island  has  various  natural  and  man-made  features,  including 
cities,  an  airport,  bridges,  roads,  and  a  volcano.  Because  of  an  expected  uprising,  the 
citizens  need  to  be  evacuated.  For  this,  they  must  first  be  brought  by  truck  to  the  capital 
city,  where  the  airport  is  located.  Evacuation  can  be  complicated  by  unexpected  road 
or  bridge  closings,  either  as  a  result  of  natural  forces,  e.g.,  a  volcano,  or  hostile  human 
forces;  it  can  also  be  complicated  by  the  fact  that  the  citizens  may  be  scattered  around 
the  island,  and  must  themselves  get  to  major  cities  before  being  taken  by  truck  to  the 
capital. 

We  assume  that  the  NEO  is  to  be  planned  and  overseen  by  several  human  planners 
(typically,  we  run  DIPART  with  between  2  and  6  planning  nodes).  Each  human  planner 
is  responsible  for  a  different  component  of  the  operation;  although  the  task  may  be 
divided  in  various  ways,  we  generally  assign  each  planner  the  task  of  moving  citizens 
from  one  city  to  the  capital.  The  exact  number  of  citizens  and  their  current  location 
may  not  be  fully  known  to  each  planner.  Each  human  planner  is  assisted  by  a  DIPART 
node;  the  human  submits  goals  to  the  node,  and  can  query  the  node  for  current  status 
information.  The  nodes  are  then  responsible  for  forming  plans  to  satisfy  the  user’s  goals, 
for  coordinating  communication  with  other  planners,  and  for  alerting  the  user  to  reports 
from  agents  in  the  (simulated)  world. 
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3  Advances  in  Planning  Technology 

The  key  task  performed  by  DIPART  nodes  is  plan  generation:  human  users  input  goals, 
such  as  evacuating  a  certain  number  of  citizens  from  some  city,  and  the  DIPART  node 
generates,  dispatches,  and  monitors  the  execution  of  a  plan  to  carry  out  that  goal. 
Consequently,  a  central  focus  of  our  research  has  concerned  the  development  of  efficient 
planning  algorithms. 

3.1  Control  during  Planning 

Many  current  state-of-the-art  planners  make  use  of  partial-order  causal  link  (POOL) 
algorithms  [27,  30].  POOL  planning  involves  searching  through  a  space  of  partial  plans, 
where  the  successors  of  a  node  representing  partial  plan  P  are  defined  to  be  the  refin- 
ments  of  P.  As  with  any  search  process,  POCL  planning  requires  effective  control;  in 
POOL  planning,  search  control  has  two  components.  The  first,  node  selection,  involves 
choosing  which  partial  plan  to  refine  next.  Most  POCL  algorithms  use  best-first  search 
to  perform  node  selection.  Once  a  partial  plan  has  been  selected,  the  planner  must  then 
perform  flaw  selection,  which  involves  choosing  either  a  threat  to  resolve  (typically,  by 
promotion,  demotion,  or  separation)  or  an  open  condition  to  establish  (by  adding  a  new 
step  to  the  plan  or  adding  a  new  causal  link  to  an  existing  step).  Unless  it  is  impossible 
to  repair  the  selected  flaw,  new  nodes  representing  the  possible  repairs  are  added  to  the 
search  space. 

In  [19],  we  explored  a  flaw-selection  strategy,  the  Least-Cost  Flaw  Repair  (LCFR) 
strategy,  which  can  be  seen  as  a  generalization  of  the  DUnf  strategy  that  had  been 
proposed  by  Peot  and  Smith  [32].  In  LCFR,  we  define  the  repair  cost  of  any  flaw — 
either  threat  or  open  condition — to  be  the  number  of  nodes  generated  as  possible  repairs. 
LCFR  is  the  strategy  of  always  selecting  a  flaw  with  the  lowest  possible  repair  cost  at 
a  given  node.  LCFR  will  delay  any  threat  that  is  unforced  (repair  cost  >  1)  in  favor 
of  a  threat  that  is  forced  (repair  cost  <=  1.)  By  treating  all  flaws  uniformly,  LCFR 
also  applies  a  similar  strategy  to  open  conditions,  preferring  to  handle  open  conditions 
that  are  forced  over  open  conditions,  or  threats,  that  are  not.  Similarly,  LCFR  handles 
the  case  in  which  all  that  remain  are  unforced  threats:  the  LCFR  strategy  will  select  a 
threat  with  minimal  repair  cost. 

Our  experimental  assessment  of  LCFR  demonstrated  that  the  power  of  DUnf  does  not 
come  from  delaying  threat  repairs  per  se.  but  rather  from  that  fact  that  this  delay  has  the 
effect  of  imposing  a  partial  preference  for  least-cost  flaw  selection.  Our  experiments  also 
showed  that  extending  this  to  a  complete  preference  for  least-cost  selection,  as  in  LCFR, 
reduces  search-space  size  even  further.  Details  of  the  experiments  can  be  found  in  [19]. 
Here  we  simply  present  the  results  of  a  key  experiment,  in  which  we  compared  5  search 
strategies  on  49  test  problems  from  a  variety  of  domains.  Figure  2  plots  the  percentage 
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Figure  2:  Comparison  of  planner  search  spaces 

of  test  problems  solved  by  each  planner  with  a  fixed  number  of  nodes  examined.  (Each 
point  (x,y)  denotes  that  y%  of  the  49  test  problems  were  solved  by  examining  no  more 
than  x  nodes.)  As  can  be  seen,  the  LCFR-based  planner  outperforms  any  of  the  others, 
including  the  two  based  on  Peot  and  Smith’s  DUnf  strategy. 

As  might  be  expected,  the  benefit  of  the  LCFR  strategy  is  not  without  a  cost:  specifi¬ 
cally,  performing  least-cost  flaw  selection  can  incur  a  significant  computational  overhead. 
We  therefore  developed  QLCFR,  which  reduces  this  overhead  by  approximating  repair 
costs,  and  we  demonstrated  its  effectiveness  experimentally.  Again,  complete  details  can 
be  found  in  [19];  subsequent  work  that  builds  on  the  LCFR  approach  includes  [44,  43]. 

3.2  Cost-Directed  Planning 

The  LCFR  strategy  described  above  is  quite  effective  for  planning  problems  in  which 
alternative  solutions  to  a  planning  problem  are  considered  to  be  roughly  equal — an 
assumption  that  is,  in  fact,  made  in  much  of  the  plan  generation  literature.  In  many 
domains,  however,  this  assumption  is  not  warranted:  for  any  given  planning  problem, 
some  solutions  have  lower  execution  cost,  some  are  more  likely  to  succeed,  and  so  on.  To 
handle  such  cases,  we  developed  a  “cost-directed”  heuristic  planner,  which  is  capable  of 
finding  low-cost  plans.  The  algorithm  performs  POCL  planning,  using  an  A*  strategy 
for  node  selection.  The  heuristic  evaluation  function  is  computed  by  a  deep  lookahead 
that  calculates  the  cost  of  complete  plans  for  a  set  of  pre-defined  top-level  subgoals, 
under  the  (generally  false)  assumption  that  those  subgoals  do  not  interact.  In  our  work 
so  far,  we  have  assumed  that  flaw  selection  is  performed  randomly,  leaving  it  to  future 
work  to  explore  the  question  of  which  flaw  selection  strategies  can  best  be  integrated 
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into  our  approach. 

In  [9],  we  show  that  the  cost- directed  planning  algorithm  not  only  leads  to  finding 
lower-cost  plans,  but  in  many  circumstances  it  does  this  without  negatively  impacting 
the  efficiency  of  planning:  in  fact,  it  can  lead  to  a  significant  decrease  in  total  planning 
time.  This  result  is  due  in  part  to  the  fact  that  generating  plans  for  a  set  of  independent 
subgoals  is  exponentially  less  costly  than  generating  a  complete  plan  taking  interactions 
into  account [23].  At  least  in  the  limit,  the  cost  of  forming  plans  for  subgoals  treated 
independently  does  not  significantly  effect  the  computational  complexity  of  the  complete 
planning  problem.  Moreover,  while  focusing  on  lower-cost  plans,  the  heuristic  function 
effectively  prunes  the  search  space.  Thus,  the  use  of  the  deep  evaluation  in  node  selection 
can  outweigh  the  marginal  additional  complexity.  Our  experiments  demonstrate  that 
the  advantages  of  cost-directed  planning  increase  with  the  complexity  of  the  planning 
problem,  where  this  is  measured  in  terms  of  the  amount  of  subgoal  interdependence,  the 
heterogeneity  of  the  cost  of  actions,  the  average  branching  factor,  and  the  number  of 
subgoals  and  length  of  the  minimal-cost  plan. 

3.3  Constraint-Based  Planning 

Both  of  the  strategies  for  controlling  planning  described  above  build  directly  on  a  tradi¬ 
tional  POOL  style  of  planning.  To  a  large  extent,  POOL  planning  was  originally  moti¬ 
vated  by  an  observation  of  the  advantages  of  taking  a  “least-commitment”  approach  to 
planning.  Least-commitment  planning  involves  postponing  decisions  until  constraints 
force  them  to  be  made.  Any  decision  made  when  it  is  not  forced  is  an  “early  com¬ 
mitment.”  POOL  planning,  as  opposed  to  the  earlier,  state-spaced  planning,  made  it 
possible  to  take  a  least-commitment  approach  to  some  decisions,  particularly  to  the 
ordering  of  plan  steps.  However,  POCL  planners  continue  to  rely  to  some  degree  on 
early  commitments  for  other  decisions,  including  variable  binding,  threat  resolution, 
and  choice  of  an  operator  to  satisfy  open  conditions. 

Because  the  least-commitment  approach  has,  by  and  large,  been  successful  where  it 
has  been  tried,  an  obvious  question  is  whether  the  least-commitment  approach  should  be 
applied  to  every  planning  decision;  in  other  words,  is  early  commitment  ever  a  good  idea? 
An  obstacle  to  addressing  this  question  experimentally  arises  from  the  way  in  which 
POCL  planners  manage  decision-making.  They  take  what  we  call  a  passive  postpone¬ 
ment  approach,  choosing  one  decision  at  a  time  to  focus  on,  and  keeping  all  the  other, 
postponed  decisions  (about  how  to  achieve  certain  goals  and  how  to  resolve  threats) 
on  an  “agenda,”  where  they  play  no  role  in  the  plan  generation  process  until  they  are 
selected  for  consideration.  The  items  on  the  agenda  may  in  fact  impose  constraints 
on  the  plan  being  generated,  but  these  constraints  are  not  available  to  the  planning 
algorithm  so  long  as  the  items  remain  on  the  agenda.  The  fact  that  constraints  exist 
but  are  not  always  accessible  makes  it  difficult  if  not  impossible  for  a  POCL  planner  to 
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be  made  more  “least  commitment”.  Postponing  decisions  until  they  are  forced  implies 
being  able  to  recognize  whether  any  decision  is  forced,  and  this  in  turn  implies  that  all 
the  constraints  that  might  affect  a  decision  must  be  available  to  (and  must  be  used  by) 
the  planning  algorithm. 

In  response  to  these  difficulties,  we  developed  a  new  approach  to  planning,  called 
active  postponement ,  in  which  even  postponed  decisions  play  a  role  by  constraining  the 
plan  being  generated.  This  technique  has  been  implemented  in  the  Descartes  system. 
The  key  idea  in  Descartes  is  to  transform  planning  problems  into  Constraint  Satisfac¬ 
tion  Problems  (CSPs)  which  can  then  be  solved  by  applying  both  planning  and  CSP 
techniques.  In  general,  a  planning  problem  cannot  be  transformed  into  a  single  static 
CSP,  however;  instead  it  must  be  transformed  into  a  dynamic  CSP  to  which  new  con¬ 
straints  and  variables  can  be  added  during  the  solution  process.  The  dynamic  CSP  is 
then  solved  by  breaking  it  down  into  static  CSPs,  to  which  standard  CSP  techniques 
may  be  applied. 

As  with  the  approaches  discussed  above,  we  have  conducted  a  number  of  experiments 
to  explore  the  power  of  constraint-based  planning.  These  experiments  demonstrate  that 
passive  postponement — even  “smart”  passive  postponement,  using  a  selection  strategy 
like  LCFR — can  result  in  significant  performance  penalties.  Further  experiments  show 
that  it  is  worthwhile  to  extend  the  least-commitment  approach  much  further  than  has 
been  done  in  prior  work.  These  results  also  suggest,  however,  that  there  are  some 
fundamental  limits  to  the  effectiveness  of  the  least-commitment  approach,  and  that 
sometimes  early  commitments  can  increase  planning  efficiency.  We  have  proposed  a 
principled  approach  to  deciding  when  to  make  early  commitments  in  planning,  based 
on  an  analysis  of  the  ongoing  constraint  processing:  specifically,  early  commitment  is 
needed  when  the  planning  process  is  forced  to  make  what  we  call  unrestricted  expansions. 
Details  of  the  constraint-based  planning  approach,  its  implementation  in  Descartes,  and 
the  results  of  experiments  using  Descartes  can  be  found  in  [20,  21,  18]. 
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4  Advances  in  Control  of  Reasoning 

As  noted  above,  planning  in  DIPART  occurs  in  a  dynamic  environment;  often,  one 
planning  problem  will  have  to  be  interrupted  so  that  attention  can  be  given  to  another 
planning  problem.  A  central  focus  of  the  DIPART  project  has  thus  been  the  development 
and  assessment  of  alternative  strategies  for  meta-level  reasoning,  i.e.  deciding  how  to 
allocate  computational  resources.  Within  the  DIPART  system  this  task  is  performed  by 
the  LMC.  The  LMC  must  decide  what  to  do  from  messages  that  can  arrive  from  four 
different  sources: 

1.  the  human  user,  who  posts  a  new  goal  to  the  system  or  tells  the  system  a  new  fact. 

2.  other  nodes,  which  may  be  seeking  information,  or  may  have  information  to  share, 
or  may  have  goals  that  they  would  prefer  to  be  handled  by  someone  else. 

3.  agents  situated  in  the  simulated  world,  who  may  transmit  a  message  to  their 
supervising  agent  (i.e.  DIPART  node)  to  report  an  unexpected  change  in  the 
environment. 

4.  reasoning  modules  within  the  node  itself,  which  post  messages  identifying  infor¬ 
mation  about  tasks  that  are  in  need  of  further  processing  by  other  RMs. 

4.1  Process  Scheduling  Algorithms  for  Meta-Level  Control 

The  problem  of  allocating  reasoning  resources  is  sometimes  called  the  deliberation- 
scheduling  problem .  Previous  approaches  to  deliberation  scheduling  in  AI  include  the 
use  of  off-line  allocation  of  on-line  deliberation  time  for  tasks  with  known  computational 
demands  [14,  2,  48],  and  the  application  of  decision-theoretic  estimations  of  optimal 
computational  sequences  [41].  Heuristic  strategies  have  been  proposed  as  well  [34]. 

The  deliberation-scheduling  problem  bears  a  strong  similarity  to  the  problems  of  pro¬ 
cess  scheduling  in  real-time  operating  systems  [45],  job  scheduling  in  operations  research 
[33],  and  transmission  scheduling  in  local  area  networks  [28].  Not  all  process-  or  job-  or 
transmission-scheduling  algorithms  are  applicable  to  deliberation  scheduling,  however. 
In  particular,  we  require  scheduling  algorithms  that  are: 

•  on-line ,  i.e.,  construct  schedules  at  run  time; 

•  dynamic ,  i.e.,  support  the  random  arrival  of  tasks; 

•  stochastic ,  i.e.,  support  tasks  with  random  computation  times;  and 

•  soft  real-time ,  i.e.,  support  the  scheduling  of  tasks  that  yield  less  than  maximal 
value  if  completed  after  some  critical  period. 
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Two  simple  and  well- re  searched  scheduling  algorithms  are  Earliest  Deadline  First 
(EDF)  and  Least  Slack  First  (LSF).  These  both  incorporate  deadline  information  and 
consequently  achieve  better  results  than  algorithms  that  do  not,  such  as  First-in-first-out 
(FIFO)  and  Round-robin  [17,  28].  It  is  known  that,  for  a  schedulable  set  of  processes, 
i.e.,  one  for  which  there  exists  an  optimal  schedule  where  all  deadlines  can  be  met,  EDF 
and  LSF  produce  a  schedule  that  meets  all  deadlines,  and  hence  performs  optimally  [1]. 
However,  the  performance  of  these  two  algorithms  degrades  sharply  when  the  system  is 
saturated ,  i.e.,  it  has  to  deal  with  a  non-schedulable  set  of  tasks. 

To  schedule  saturated  job  sets  effectively,  scheduling  algorithms  must  take  into  ac¬ 
count  the  cost  of  missing  a  deadline.  This  is  particularly  true  when  there  are  trade-offs  in 
the  acceptance  rate  and  the  deadline  miss  rate  of  tasks  in  the  system.  The  environments 
of  autonomous  agents  typically  present  such  trade-offs:  it  may  well  be  worth  missing 
the  deadlines  for  some  tasks  in  order  to  achieve  higher- quality  performance  on  other 
tasks.  Such  trade-offs  can  be  evaluated  with  the  aid  of  value-density  assessment  tools. 
The  value  density  of  any  task  t  is  defined  to  be  the  value  to  the  system  of  completing  t 
divided  by  its  remaining  computation  time.  Value-density  assessments  are  included  in 
scheduling  algorithms  such  as  Best-Effort  (BE)  and  Dynamic-priority  (DP);  previous  re¬ 
search  has  shown  that  these  algorithms  perform  better  than  EDF  and  LSF  in  saturated 
environments  [17,  28]. 

We  have  explored  the  usefulness  for  deliberation  scheduling  of  the  value-density  mea¬ 
sure  and  the  algorithms  that  rely  on  it.  Specifically,  we  identified  appropriate  candidate 
algorithms,  conducted  preliminary  experiments  to  compare  their  performance  of  these 
algorithms,  demonstrated  a  proof-of-concept  use  of  these  algorithms  for  deliberation 
scheduling  in  the  DIPART  system,  and  analyzed  current  limitations  of  the  proof-of- 
concept  system,  i.e.,  identified  certain  assumptions  that  are  made  in  the  existing  algo¬ 
rithms  that  must  be  relaxed  to  support  full-fledged  deliberation  scheduling.  In  addition, 
we  developed  a  modification  of  the  Best  Effort  algorithm  that  results  in  improved  per¬ 
formance  for  the  DIPART  job  mix.  Details  can  be  found  in  [39,  40]. 

4,2  Contingency  Selection  in  Plan  Generation 

Classical  AI  plan  generation  systems  assume  static  environments  and  omniscient  agents, 
and  thus  ignore  the  possibility  that  events  may  occur  in  unexpected  ways — that  con¬ 
tingencies  might  arise — during  plan  execution.  The  plans  these  systems  construct  are 
simply  sequences  of  actions:  they  assume  that  the  state  that  will  occur  after  action  is 
taken  is  the  one  in  which  A{+ 1  should  be  performed.  The  problem  with  classical  planners 
is,  of  course,  things  do  not  always  go  “according  to  plan.” 

In  contrast,  universal  planning  systems  [42]  and  more  recent  Markov-decision  process- 
based  systems  (e.g.,  [7,  3])  make  no  such  assumptions.  They  produce  “plans”  or  “poli¬ 
cies”  that  are  functions  from  states  to  actions.  After  performing  an  action  in  one  state, 
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the  agent  then  looks  to  see  what  new  state  it  is  in,  and  performs  the  action  prescribed 
for  that  state.  Effectively,  every  contingency  is  handled  at  planning  time.  The  problem 
with  universal  planning  and  MDP  systems  is  that  the  state  space  is  typically  much  too 
large  for  them  to  be  effective. 

Conditional  planners  take  the  middle  road.  They  do  not  assume  that  the  result  of 
each  action  taken  is  necessarily  known  at  planning  time;  instead  they  allow  for  both 
conditional  actions  with  multiple  possible  contingencies,  and  for  sensing  actions  that  al¬ 
low  agents  to  determine  which  contingency  occurred  (e.g.,  [8,  31,  37])  A  key  question  in 
conditional  planning  is:  how  many,  and  which  contingencies  should  be  selected  so  that 
the  plan  can  be  extended  to  include  actions  that  will  be  taken  in  case  the  contingency 
fails?  One  cannot  plan  for  all  possible  failures,  or  one  will  inherit  the  problems  of  uni¬ 
versal  planners.  One  cannot  ignore  all  possible  failures,  or  one  will  inherit  the  problems 
of  classical  planners. 

The  need  to  select  contingencies  was  observed  by  Peot  and  Smith  [31,  p.  196],  who, 
in  describing  the  conditional  planner  CNLP,  noted  that  “the  size  and  complexity  of 
the  plans  generated  by  CNLP  increase  exponentially  with  the  number  of  observation 
actions  in  the  plan.  The  amount  of  computation  may  be  reduced  by  attaching  a  relative 
likelihood  measure  to  the  various  contexts  in  the  plan  . . .  and  skip  [ing]  contexts  that  are 
sufficiently  unlikely  . . .  ” .  We  have  developed  a  planning  algorithm  that  generalizes  this 
suggestion,  by  deciding  which  of  the  possible  failures  to  plan  for,  based  not  just  on  their 
likelihood  of  occurrence,  but  also  on  the  damage  they  would  cause  if  in  fact  they  were  to 
occur.  Put  otherwise,  our  algorithm  attempts  to  identify  the  influence  each  contingency 
would  have  on  the  outcome  of  plan  execution,  and  then  proceeds  by  giving  priority  to 
the  contingencies  whose  failure  would  have  the  greatest  negative  impact. 

We  define  contingencies  to  be  alternative  outcomes  of  probabilistic  actions,  and  our 
approach  is  to  directly  reason  about  the  value  of  planning  for  the  failure  of  various  con¬ 
tingencies.  In  order  to  make  this  feasible,  we  have  adopted  a  different  strategy  towards 
conditional  planning  than  that  taken  in  some  earlier  systems,  notably  C-BUREDAN  [8], 
which  is,  to  our  knowledge,  the  only  partial-order  conditional  planner  that  does  not 
necessarily  plan  for  all  contingencies.  C-BURLDAN  constructs  branches  for  alternative 
contingencies  in  a  somewhat  indirect  fashion.  It  sometimes  discovers  that  there  are  two 
incompatible  actions,  say,  Al  and  A2,  each  of  which  can  achieve  some  condition  C.  To 
resolve  the  threats  between  A1  and  A2,  C-BURIDAN  introduces  an  observation  action  O 
with  two  contingent  outcomes,  and  then  “splits”  the  plan  into  two  branches,  associating 
each  outcome  with  one  of  the  actions.  In  the  process  of  doing  this,  C-BURIDAN  does 
not  reason  about  whether  both  those  contingent  outcomes  are  worth  planning  for. 

Our  algorithm  works  by  iterative  refinement.  It  first  finds  a  skeletal  plan  to  achieve 
the  goals,  and  then  during  each  iteration  selects  a  contingency  whose  failure  will  have  a 
maximal  disutility ,  i.e.,  one  that  will  have  a  maximal  negative  impact  if  it  does  not  occur. 
The  algorithm  then  extends  the  plan  to  include  actions  to  take  in  case  the  contingency 
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fails.  Iterations  proceed  until  the  expected  utility  of  the  plan  exceeds  some  specified 
threshold.  Details  of  this  work  can  be  found  in  [29]. 


12 


5  Advances  in  Distributed  Planning  and  Commu¬ 
nication 

One  addition  research  topic  that  centrally  concerns  us  is  the  specification  of  appropriate 
communication  and  coordination  strategies  for  multi-agent,  dynamic  planning.  We  first 
briefly  describe  the  communication  package  we  have  implemented  to  support  communi¬ 
cation  among  DIPART  nodes,  and  then  sketch  approaches  to  multi-agent  planning  that 
we  have  been  investigating. 

5.1  Software  Support  for  Communication 

Communication  among  nodes  in  DIPART  is  built  on  a  group  management  model  [6]. 
Groups  of  processes  (or,  in  our  case,  agents)  cluster  into  a  single  logical  entity  such 
that  all  communications  sent  by  a  member  of  the  group  are  received  by  all  members  of 
the  group.  Thus  multi-process  communication  is  achieved  by  a  single  operation  rather 
than  by  a  series  of  operations  to  a  (potentially  unknown  or  partially  known)  set  of 
individual  agents.  Group  operations  can  take  advantage  of  network-multicast  capability, 
thus  reducing  communication  overhead  and  increasing  concurrency. 

Using  a  group  management  model,  we  have  implemented  a  set  of  communication 
primitives  that  enable  the  basic  group  operations  (e.g.,  form  a  group,  dissolve  a  group, 
join  a  group,  leave  a  group,  invite  to  a  group,  and  exclude  from  a  group)  and  commu¬ 
nication  actions  (e.g.,  send,  send  and  block,  receive,  group-cast,  group-cast  and  block, 
receive  any,  receive  any  and  block).  Groups  may  have  different  structures,  which  de¬ 
termine  the  relationship  among  group  members.  In  a  coordinated  group ,  the  owner  of 
the  group  must  approve  any  new  members,  while  in  peer  groups,  all  new  members  are 
accepted.  There  are  also  different  group  types:  in  a  private  group,  communication  is 
restricted  only  to  the  members  of  the  group,  while  non- members  may  send  messages  to 
public  groups.  We  have  also  implemented  a  group  server,  which  maintains  information 
about  the  status  and  membership  of  each  group,  and  is  responsible  for  synchronizing 
group  actions.  Additional  details  can  be  found  in  [49,  24], 

5.2  Load  Balancing  for  Distributed  Planning 

The  communications  package  can  be  used  to  support  a  process  of  load  balancing  among 
the  DIPART  agents,  so  that  no  agent  falls  behind  as  a  result  of  having  too  many  re¬ 
sponsibilities,  while  other  agents  sit  idle.  We  have  investigated  a  range  of  load-balancing 
techniques  developed  in  the  distributed  operating-systems  literature,  focusing  in  partic¬ 
ular  on  those  that  use  dynamic  thresholds. 

The  purpose  of  dynamic  thresholds  is  to  give  the  agent  more  flexibility  to  adapt  itself 
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to  a  changing  environment.  Consider  a  gift  shop  as  an  example:  if  the  shop  receives  a 
hundred  customers  in  a  regular  week  day  it  is  a  busy  day.  However,  if  the  shop  receives 
a  hundred  customers  just  before  Christmas,  it  is  not  a  busy  day.  Similarly,  in  a  mili¬ 
tary  application,  one  would  expect  higher  processing  loads  during  crisis  situations  than 
during  routine,  peacetime  operations.  Instead  of  determining  a  priori  what  is  a  high 
load,  dynamic  load  balancing  evaluates  the  load  of  an  agent  at  running  time  according 
to  the  partial  information  it  possesses  about  the  environment.  As  a  consequence,  given 
the  same  amount  of  tasks  to  perform  the  same  agent  may  consider  itself  highly  loaded 
or  lightly  loaded  depending  on  its  estimation  of  the  system  load.  Dynamic  thresholds 
are  suitable  in  dynamic  environments  when  a  system  must  avoid  unnecessary  communi¬ 
cations  that  would  add  an  extra  overhead  to  an  already  overloaded  system. 

Another  way  to  lessen  communication  load  is  to  employ  selective  unicasting.  In 
load  balancing,  one  faces  a  trade-off  between  the  cost  of  exchanging  messages  and  the 
necessity  of  having  an  information  accurate  enough  to  provide  efficient  load  balancing. 
When  initiating  load  balancing  an  agent  could  send  a  message  to  every  other  agent  asking 
information  about  their  loads  and  wait  for  the  answers  before  selecting  the  best  agent 
to  balance  with.  However  this  classical  scheme  has  two  drawbacks:  it  requires  many 
messages  to  be  exchanged,  and  it  is  not  fault -tolerant — it  does  not  include  provisions 
for  the  case  in  which  one  or  more  agents  is  unable  to  respond.  In  contrast,  in  selective 
unicasting  the  messages  concerning  the  exchange  of  information  about  the  load  of  the 
system  are  piggybacked  to  task  balancing  messages  and  therefore  induce  almost  no 
overhead.  Also,  the  scheme  is  non-blocking,  and  will  not  collapse  should  one  or  more 
agents  fail.  However,  selective  unicasting  with  piggybacking  results  in  the  agent  being 
unable  to  access  complete  information  about  the  system’s  current  load.  Thus,  each  agent 
must  estimate  this  information  by  using  data  about  the  previous  load  history. 

The  load-balancing  algorithms  we  constructed  were  implemented  and  subjected  to 
experimentation  to  assess  their  performance  relative  to  a  variety  of  alternatives,  including 
a  broadcasting  scheme;  we  also  studied  the  relative  effectives  of  client-driven,  server- 
driven,  and  hybrid  variants.  We  measured  two  things:  throughput  and  efficiency ,  both 
of  which  were  defined  in  terms  of  the  Pacifica  scenario.  For  instance,  throughput  was 
taken  to  be  the  ratio  p/d,  where  p  is  the  number  of  passengers  for  whom  transportation 
has  been  requested,  and  d  is  the  delay  between  the  time  of  the  first  goal  submitted  and 
the  completion  of  the  last  request.  The  throughput  is  given  by  the  ratio  p/d. 

Details  of  the  experiments,  and  complete  results,  can  be  found  in  [25,  26].  The  most 
important  result  is  that  selective  load  balancing  (hybrid)  yields  a  good  throughput, 
even  compared  with  load  balancing  with  broadcast.  When  compared  with  the  lower 
bound,  hybrid  load  balancing  achieves  a  performance  34%  higher,  client  driven  load 
balancing  achieves  a  performance  15%  higher,  and  server  driven  load  balancing  achieves 
a  performance  11%  higher.  Compared  to  the  upper  bound,  selective  load  balancing 
performs  only  7%  worse,  while  using  many  fewer  messages.  Thus  it  appears  possible 
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to  achieve  effective  load  balancing  by  using  dynamic  thresholds,  even  if  communication 
must  be  minimized. 


5.3  Plan  Merging 

The  load-balancing  work  involves  agents  sharing  the  work,  but  each  individually  forming 
their  own,  more  or  less  complete  plans.  Sometimes  this  is  feasible,  but  at  other  times, 
agents  need  to  form  partial  plans,  which  are  then  merged  together.  We  identified  four 
different  types  of  situations  in  which  some  merging  may  be  needed.  In  the  first,  a  group 
of  agents  has  to  cooperatively  achieve  one  common  global  goal.  In  the  second  type  of 
situation,  due  to  time  constraints,  execution  of  the  plan  is  interleaved  with  the  planning 
process  itself.  In  the  third  third,  each  agent  has  its  own  private,  individual  goal.  There 
is  also  a  fourth  situation,  in  which  planning  and  execution  are  interleaved  for  a  group 
of  agents  with  private  goals.  The  DIPART  scenario  can  be  viewed  either  as  an  instance 
of  the  second  of  the  fourth  type,  depending  on  how  much  knowledge  each  of  the  human 
planning  agents  has  about  the  plan  for  the  overall  mission. 

For  each  of  these  situations,  we  described  how  a  global  plan  is  constructed  through  the 
process  of  incrementally  merging  sub-plans.  By  making  use  of  the  computational  power 
of  multiple  agents  working  in  parallel,  the  process  is  able  to  reduce  the  total  elapsed 
time  for  planning  as  compared  to  a  central  planner.  For  the  case  in  which  agents  do  not 
have  complete  knowledge  of  the  overall  mission,  we  show  how  agents  can  reach  consensus 
about  what  multi-agent  plan  to  carry  out  using  a  voting  procedure,  without  having  to 
reveal  full  goals  and  preferences  (unless  that  is  actually  necessary  for  consensus  to  be 
reached).  Our  technique  also  does  away  with  the  need  to  generate  final  alternatives 
ahead  of  time  (instead,  candidate  states  arise  at  each  step  as  a  natural  consequence  of 
the  emerging  plan).  The  agents  iteratively  converge  to  a  plan  that  brings  the  group  to 
a  state  maximizing  the  overall  system  utility.  Details  and  experimental  results  can  be 
in  [11,  10,  13]. 

5.4  Multi- Agent  Filtering 

In  addition  to  plan  merging,  which  involves  explicit  coordination  among  agents,  it  is 
sometimes  useful  for  agents  to  have  a  means  of  achieving  coordination  implicitly.  We 
have  been  investigating  a  strategy  for  implicit  coordination  called  multi-agent  filtering. 
It  extends  a  single-agent  strategy,  filtering,  which  was  developed  as  a  way  of  controlling 
reasoning  in  dynamic  environments.  The  notion  of  single-agent  filtering  derives  from 
the  work  of  Bratman  [4];  it  involves  an  agent  committing  to  the  goals  it  has  already 
adopted,  and  tending  to  bypass  (or  “filter  out”)  new  options  that  would  conflict  with 
their  successful  completion  [5,  34,  35].  We  and  others  have  studied  the  effectiveness  of 
filtering  in  domains  with  various  characteristics [36,  22,  35]. 
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Where  single-agent  filtering  means  tending  to  bypass  options  that  are  incompatible 
with  an  agent’s  own  goals,  multi-agent  filtering  means  tending  to  bypass  options  that  are 
incompatible  with  any  agent’s  known  or  presumed  goals.  We  examined  several  forms  of 
multi-agent  filtering,  which  range  from  purely  implicit,  in  which  agents  have  rules  of  legal 
action  that  lead  to  their  avoiding  conflict  without  ever  reasoning  explicitly  about  one 
another’s  goals,  to  minimally  explicit,  in  which  agents  perform  very  shallow  reasoning 
to  assess  whether  their  actions  are  incompatible  with  the  likely  intended  actions  of  other 
agents.  In  no  cases  do  the  agents  engage  in  any  explicit  negotiation. 

Our  experimental  results  on  the  efficacy  of  multi-agent  filtering  are  presented  in 
[12].  The  most  interesting  and  surprising  result  is  that,  at  least  for  the  simple,  abstract 
environments  so  far  bstudied,  multi-agent  filtering  is  a  dominant  strategy:  no  matter 
what  proportion  of  the  agents  in  some  environment  choose  not  to  filter,  those  that  do 
filter  perform  better. 
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6  Conclusions 


The  traditional  planning  paradigm  in  the  AI  literature  has  made  a  number  of  extremely 
strong  assumptions,  including  the  following: 

1.  There  is  a  single  planning  agent,  who  is  the  only  cause  of  change  in  the  environ¬ 
ment. 

2.  The  planning  agent  knows  all  the  relevant  facts  about  its  environment. 

3.  The  goals  presented  to  the  agent  remain  unchanged  throughout  the  process  of 
planning  and  execution. 

4.  The  goals  are  categorical,  i.e.,  they  are  fully  achieved  or  not  (there  is  no  notion  of 
partial  satisfaction). 

5.  The  actions  to  be  performed  have  certain  outcomes. 

Of  course,  these  assumptions  are  violated  in  most  real-world  applications.  In  a  crisis- 
action  setting,  for  example,  there  will  typically  be  many  planning  agents,  each  with 
incomplete  information  about  the  environment,  who  must  coordinate  with  one  another 
in  the  formation  and  execution  of  plans.  The  world  will  change  not  only  as  a  result  of 
these  agents’  actions,  but  also  as  a  result  of  the  actions  of  other  agents,  who  may  be 
hostile;  changes  may  also  occur  independent  of  any  agent’s  actions,  for  example,  as  a 
result  of  weather  or  other  natural  activity.  As  the  situation  develops,  new  goals  may 
arise,  and  previous  goals  may  become  irrelevant.  The  actions  to  be  performed  will  have 
uncertain  outcomes,  and  it  will  be  possible  to  achieve  some  goals  only  partially. 

The  DIPART  project  has  been  concerned  with  expanding  the  planning  process,  to 
weaken  the  first  three  assumptions.  (We  have  been  less  concerned  with  the  fourth  and 
fifth  assumptions;  but  see  the  growing  body  of  work  on  decision-theoretic  planning,  e.g. 
[15,  47,  16].)  We  have  made  progress  towards  our  goal  of  supporting  automated  plan¬ 
ning  in  dynamic,  multi-agent  settings,  by  extending  our  understanding  of  the  algorithms 
that  can  support  planning.  Thus,  as  described  in  this  report,  we  have  developed  better, 
more  efficient  techniques  for  plan  generation,  for  controlling  the  planning  process,  and 
for  coordinating  planning  processes  among  multiple  agents.  In  developing  these  tech¬ 
niques,  we  drew  on  earlier  research  both  from  real-time  and  distributed  AI,  and  from 
real-time  and  distributed  operating  systems.  Moreover,  we  employed  an  experimental 
methodology,  in  which  we  explored  the  performance  of  our  proposed  algorithms  under 
different  conditions  by  conducting  systematic  experiments  using  experimental  platforms, 
including  the  DIPART  simulator. 

Of  course,  many  research  questions  remain  unanswered.  In  our  view,  perhaps  the 
most  important  involves  further  development  of  the  foundations  of  dynamic  planning: 
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planning  in  situations  in  which  one’s  goals  and  knowledge  change,  sometimes  dramat¬ 
ically,  over  time.  In  particular,  little  has  yet  been  done  on  the  question  of  exploiting 
one’s  expectations  about  future  change  during  the  planning  process.  For  example,  if  a 
planning  agent  expects  that  it  will  soon  gain  a  large  amount  of  additional  information, 
it  may  choose  to  defer  planning  and/or  to  form  only  highly  abstract  plans  until  that 
additional  information  is  received.  A  related  problem  involves  forming  plans  that  are 
robust  with  respect  to  anticipated  changes  in  the  environment.  A  plan  that  would  be 
optimal  if  the  environment  does  not  change  might  well  be  suboptimal  in  light  of  changes 
that  can  reasonably  be  expected  to  occur.  Current  planning  technology  does  not  include 
mechanims  for  reasoning  about  expected  change. 

Another  critical  question  still  to  explored  is  the  cost  of  learning  in  dynamic  environ¬ 
ments.  Just  as  there  is  a  trade-off  between  communicating,  to  receive  more  informa¬ 
tion,  and  acting  on  the  basis  of  what  is  already  known,  in  a  dynamic  environment,  an 
agent  must  weigh  the  cost  of  learning  a  new  procedure  for  achieving  a  goal  against  the 
continued  use  of  an  already  known,  but  possibly  less  efficient  procedure.  Complete  de¬ 
liberation  about  the  utility  of  learning  is  not  always  feasible  in  a  dynamic  environment; 
more  efficient,  if  suboptimal,  strategies  for  determining  when  to  attempt  to  learn  must 
be  designed.  The  development  of  such  strategies  awaits  further  investigation. 
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MISSION 

OF 

ROME  LABORA  TOR  Y 


Mission.  The  mission  of  Rome  Laboratory  is  to  advance  the  science  and 
technologies  of  command,  control,  communications  and  intelligence  and  to 
transition  them  into  systems  to  meet  customer  needs.  To  achieve  this, 
Rome  Lab: 

a.  Conducts  vigorous  research,  development  and  test  programs  in  all 
applicable  technologies; 

b.  Transitions  technology  to  current  and  future  systems  to  improve 
operational  capability,  readiness,  and  supportability; 

c.  Provides  a  full  range  of  technical  support  to  Air  Force  Material 
Command  product  centers  and  other  Air  Force  organizations; 

d.  Promotes  transfer  of  technology  to  the  private  sector; 

e.  Maintains  leading  edge  technological  expertise  in  the  areas  of 
surveillance,  communications,  command  and  control,  intelligence, 
reliability  science,  electro-magnetic  technology,  photonics,  signal 
processing,  and  computational  science. 

The  thrust  areas  of  technical  competence  include:  Surveillance, 
Communications,  Command  and  Control,  Intelligence,  Signal  Processing, 
Computer  Science  and  Technology,  Electromagnetic  Technology, 
Photonics  and  Reliability  Sciences. 


